1. Field of the Invention
This invention relates to copy operations between a primary storage volume and one or more data mirror volumes. More particularly, the invention relates to maintaining data consistency between a synchronous data mirror volume and an asynchronous data mirror volume when the primary storage volume becomes unavailable.
2. Description of the Related Art
It is well known that a CPU randomly and sequentially updates one or more data storage volumes in an attached storage subsystem. It is further known that remote electronic copying of data storage volumes is a frequently used strategy for maintenance of continuously available information systems in the presence of a fault or failure of system components. Among several copy techniques, mirroring is often favored over point-in-time copying because a data mirror may be quickly substituted for an unavailable primary volume.
Conventionally, volume-to-volume mirroring from a primary volume to a data mirror volume is accomplished either synchronously or asynchronously. Synchronous mirroring can be made transparent to applications on the central processing unit (CPU) and incur substantially no CPU overhead by direct control unit to control unit copying. However, completion of a write or update is not given to the host until the write or update is completed at both the primary mirror volume and the synchronous mirror volume. In contrast, asynchronous mirroring allows the CPU access rate of the primary volume to perform independent of the mirror copying. The CPU may, however, incur copy management overhead.
In recognition for the need of multiple backup/copy approaches, several large systems offer a suite of copy functions. One such suite is offered as part of the IBM Enterprise Storage Server (ESS) package. This package includes synchronous volume-to-volume copy operations under the control of a storage controller, for example, the Peer-to-Peer Remote Copy (PPRC). It also includes asynchronous single or multi-volume copying under host control such as the Extended Remote Copy (XRC).
Referring now to FIG. 1, a prior art peer-to-peer remote copy (PPRC) system 100 is illustrated. The PPRC system 100 exemplifies a synchronous mirror system and includes a primary storage system 110 and a secondary storage system 120. A primary host 130 is connected to the primary storage system 110. The primary host 130 stores data by sending write requests to the primary storage system 110.
Data written to primary storage system 110 is copied to the secondary storage system 120, creating a mirror of the data on the primary storage system 110 on the secondary storage system 120. The copy process is a synchronous data mirroring process. In the PPRC system 100, a write made by primary host 130 is considered complete only after the data written to the primary storage system 110 is also written to the secondary storage system 120. The primary host 130 may take various forms, such as a server on a network, a Web server on the Internet, or a mainframe computer. The primary storage system 110 and secondary storage system 120 are disk systems in these examples.
A communication path 140 connects the primary host 130 to the primary storage system 110. A communication path 150 connects the primary storage system 110 with the secondary storage system 120. Communication paths 140 and 150 may comprise various links, such as fiber optic lines, packet switched communication links, enterprise systems connection (ESCON) fibers, small computer system interface (SCSI) cable, and wireless communication links.
The primary storage system 110 includes at least one storage volume 160 typically referred to as a primary volume and other well-known components such as a controller, cache, and non-volatile storage. The secondary storage system 120 includes at least one storage volume 170, typically referred to as a secondary volume. The volumes 160, 170 in the primary and secondary storage systems 110, 120 are set up in PPRC pairs. PPRC pairs are synchronous mirror sets in which a storage volume in the primary storage system 110 has a corresponding storage volume in the secondary storage system 120. For instance, primary storage volume 160 is paired with secondary storage volume 170. This pair is referred to as an established PPRC pair or synchronous mirror set, wherein the secondary storage volume 170 mirrors the data on the primary storage volume 160.
In operation, each time a write request is sent to the primary volume 160 by the primary host 130, the primary storage system 110 stores the data on the primary volume 160 and also sends the data over the communication path 150 to the secondary storage system 120. The secondary storage system 120 then copies the data to the secondary volume 170 to form a mirror of the primary volume 160. In some systems, a non-volatile cache in the primary storage system 110 and/or a non-volatile cache in the secondary storage system 120 may be temporarily used to store data directed at the primary storage volume 160 and/or the secondary storage volume 170.
Significantly, the primary storage system 110 must receive an acknowledgement that the copied data has been written to the secondary storage system 120 before terminating the I/O operation. This means that a subsequent I/O access to the same block cannot start until after the acknowledgement has been received. This acknowledgement requirement increases the response time to write requests directed to the primary storage system. In addition, as the distance between the primary storage system and the secondary storage system is increased the response time is also increased, which further decreases performance. High response times can cause unacceptable latency for completing the transaction.
The asynchronous remote copy method (XRC) is an asynchronously mirrored, volume-to-volume copy process. XRC asynchronously copies track updates on a primary volume in a primary storage system to a secondary volume in a secondary storage system. The copies are often transmitted over a long-distance communications path, possibly thousands of kilometers in length.
Referring to FIG. 2, this figure depicts a prior art XRC system 200 including a primary site 210 and a secondary site 220. The XRC system 200 exemplifies an asynchronous mirror system. The primary site 210 includes a primary host 230, for example, an IBM host running DFSMS/MVS host software. The primary host 230 further includes one or more application programs 235. A primary storage system 245 is connected to the primary host 230 by one or more channels, for example, fiber optic channels. Contained within or connected to the primary storage system 245 is at least one primary volume 250.
The secondary site 220, located for example, some thousands of kilometers remote from the primary site 210, includes a secondary host 260 having a data mover 265 operating therein. A secondary storage system 270 is connected to the secondary host 260 via one or more channels. Contained within or connected to the secondary storage system 270 is at least one secondary volume 280, typically called an asynchronous mirror volume.
The primary storage system 245 communicates with the secondary site 220 via a communication link 290. More specifically, the primary storage system 245 provides data and control information to the secondary host 260 by a communications protocol. The communication link 290 can be realized by multiple suitable communication methods, including telephone (T1, T3 lines), radio, radio/telephone, microwave, satellite, etc.
The XRC system 200 encompasses collecting data from the primary storage systems 245 so that all write requests from the primary host 230 to the primary volume 250 are preserved and applied to the secondary volume 280 without significantly impacting access rates for the primary host 230. The data and control information transmitted to the secondary site 220 must be sufficient such that a consistent copy of the primary volume 250 is established at the remote site 220.
The applications 235 generate write requests which update data on the primary storage system 245. The locations of the data updates are monitored and recorded by the primary storage system 245. In addition, an array of bits, often referred to as an active track array or changed track array, is typically used to keep a real-time record by track address on the primary volume that have been changed since the last synchronization. The changed track array is maintained in the primary storage system 245.
The updates are provided by the primary storage system 245 via the communication link 290 to the data mover 265. The data mover 265 may form the updates into a consistency group and thereafter transfer the consistency group to the secondary storage system 270, which writes the updates to the secondary volume 280. To maintain data integrity during an XRC session, the primary storage system 245 may use a second bit map, sometimes called a recovery track array or copy track array, to designate which tracks are currently being copied from the primary volume 250 to the secondary volume 280. The copy track array is maintained in the primary storage system 245.
The copy track array is typically loaded with the contents of the changed track array at the start of a synchronization operation and then the changed track array is cleared, permitting the changed track array to track the subsequent write requests to the primary volume 250. The copy track array identifying tracks that the primary storage system 245 must copy to the secondary volume 280 is cleared when acknowledgement is received that the tracks have been successfully copied to the secondary volume 280.
In the event that communication is lost during a copy session with the secondary host 260, due to any reason, the copy track array indicates which tracks must be retransmitted to retry the previous synchronization of the secondary volume 280. In some implementations, the data mover 265 inspects the updates to determine whether any records for a given time interval have been lost or are incomplete.
XRC has minimal impact on the access rate between the primary host 230 and the primary storage system 245 because a subsequent I/O operation may start directly after receiving acknowledgement that data has been written to the primary volume 250. While write requests may occur constantly according to the needs of the application programs 235, the synchronization of the secondary volume 280 is an independent, asynchronous event scheduled periodically throughout the day, typically several times per minute. Thus, an asynchronous mirror volume is only rarely identical to the primary volume 250, since writes requests to the primary volume 250 may occur during the copy operation necessary to synchronize the asynchronous mirror volume.
If the changed track array becomes unavailable for any reason, the data mover 265 cannot determine the location of tracks changed since the last synchronization in order to copy the tracks to the secondary volume 280. Additionally, if the copy track array becomes unavailable during a synchronization, the data mover cannot determine the locations of tracks relating to the interrupted synchronization that remain to be copied to the secondary volume 280. Consequently, to ensure consistency, the XRC system 200 typically performs a time consuming process of copying the entire contents of the primary volumes to the associated secondary volumes to reconstruct the asynchronous mirror.
In some systems, both synchronous and asynchronous data mirrors are maintained. This configuration permits rapid promotion of a synchronous mirror system to become a replacement primary storage system in the event that the original primary storage system becomes unavailable. The configuration also provides for the maintenance of a nearly real-time remote copy of the primary storage system data for use if the primary site becomes unavailable. In this configuration, the storage volumes on the primary storage system may act as the primary volumes for both local synchronous mirror volumes and remote asynchronous mirror volumes.
However, if the primary storage system incurs a fault or becomes otherwise unavailable, the changed track array and the copy track array stored in the primary storage system become unavailable. Without access to this information, synchronization of the asynchronous mirror may require copying the contents of the complete set of primary volumes or the synchronous secondary volumes to the associated asynchronous mirror volumes. In a large installation, this synchronization may involve hundred of volumes and may require hours or days of recovery time. Since the asynchronous mirror cannot offer protection until the synchronization is complete, the system data will be unprotected against a disaster at the primary site during the recovery period.
A need exists for a method, apparatus, and system to synchronize an asynchronous mirror volume using a synchronous mirror volume. Beneficially, such a method, apparatus, and system would decrease recovery time from a primary storage system going offline and provide a means for a continuous remote copy of system data to be maintained for use in the event the primary site becomes unavailable.